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A SYSTEM AND METHOD FOR AUTHORING, DISTRIBUTING AND 
REPLAYING DERIVATIVE HYPERMEDIA CONTENT 

The present invention relates to the field of information processing on a network 
such as the World Wide Web (WWW) and, more specifically, to authoring, 
5 distributing, and replaying derivative hypermedia content. 

As the World Wide Web (WWW) becomes a more significant information source, 
there is an increasing need to share and reuse WWW information. 

It is herein recognized as being desirable that a WWW user should be enabled to 
place annotations on WWW documents. There are prior art systems which allow 

10 certain kinds of annotations to be placed on WWW documents. The well-known 

Mosaic browser allows annotation of static text to a WWW document. However, 
this system has the limitation that the annotation was stored along with the 
document, and thus required the permission of the document owner to create an 
annotation. This prevented annotation on arbitrary pages as permission was 

15 typically not forthcoming, and it also resulted in security risks for the document 

owners who did grant permission. It also prevented users from having private 
annotations on a document. 

Other systems, such as the ViewDirector Prism system, from TMS Sequoia, and 
the HotOffTheWeb system from Insight Development, allow graphical annotation 

20 on WWW documents. However, these annotations are static in nature, meaning 

that there is no timing or navigation information stored with them. Such a system 
cannot easily be used to author a tour through several pages as is herein recognized 
to be desirable. Moreover, these systems do not support voice annotation. 
A natural mechanism for explanation or exposition is the so-called dynamic 

25 annotation, such as is described for image data in U.S. Patent No. 5,838,3 13, issued 

17-Nov-1998 in the names of Hou, Tai-Yuan et al. (Siemens Docket #95P7547), 
whereof the disclosure is hereby incorporated by reference to the extent it is not 
inconsistent with the present application. Briefly, that application discloses a 
multimedia-based reporting system which utilizes a command interpreter to receive 

30 inputs from a user and forwards the inputs to an open report handler, a save report 
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handler, a delete current report handler, a mail report handler, a static annotation 
handler, a dynamic annotation handler including a recording and playback handler, 
an annotation on annotation handler, an attach media handler and a print report 
handler. Most relevantly to the current invention, the dynamic annotation handler 
5 allows the individual to record/playback annotations from the user input. 

Creating an annotation involves recording the mouse gestures, drawings, and voice 
of the annotation author, along with timing information, so that the annotation can 
be later replayed in just the way that the author created it. Unlike the Mosaic, 
Prism, and Sequoia systems mentioned above, users of a dynamic annotation 
10 system could see the graphical annotations unfold in time with the comments of the 

author. 

In accordance with an aspect of the present invention, there is described a tool 
which allows a WWW user to author a new kind of WWW content which is 
derivative from existing WWW documents. A user is able to add annotations to 
15 WWW documents, and record a "guided tour" through a series of WWW 

documents. The resulting tour is a combination of the existing WWW documents, 
the ordering and timing introduced by the author, and the author's annotations. 

In accordance with another aspect of the invention, the idea of dynamic annotation 
is extended to also record navigation events. In this way, hypermedia documents 
20 can be toured in an annotation. 

In accordance with an aspect of the present invention, there is described a 
multimedia based system for authoring, distributing, and replaying derivative 
hypermedia content, including a tool which, for example, allows a WWW user to 
author a new kind of WWW content which is derivative from existing WWW 
25 documents. 

In accordance with another aspect of the invention, there are primarily three 
components to the system: an authoring system for recording dynamic annotations 
on hypermedia, a distribution system for distribution of those annotations, and a 
playing system for playing those annotations. The authoring and playing systems 
30 are comprised in a hypermedia browser, while the distribution system is partly 
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comprised in th^^owser and partly in a special remote^erver. The part of the 
distribution system within the browser is called the annotation manager, and the 
remote server is called the annotation server. 

The present invention, as hereinafter described by way of exemplary embodiments, 
5 integrates the idea of dynamic annotation with a hypermedia browser such as a 

WWW browser. A traditional hypermedia browser is augmented with systems for 
recording, distributing, and playing dynamic annotations of hypermedia 
documents. The invention generates new opportunities for creating and sharing 
derivative content from hypermedia sources such as the WWW. 

10 The invention will be more fully understood from the following detailed 

description of the preferred embodiments, in conjunction with the drawing, in 
which 

Figure 1 shows the overall architecture of the system, as implemented for use with 
the WWW; 

15 Figure 2 shows a layout of an embodiment in accordance with the present 

invention; Figure 3 shows the flow of control of a hypermedia browser in 
accordance with the present invention; 

Figure 4 shows the operation of the system in accordance with the present 
invention in annotation mode; 

20 Figure 5, in reference to a system in accordance with the invention, shows the 

operation of the annotation distribution system for the storing of an annotation. 

Figure 6, in reference to a system in accordance with the invention, shows the 
operation of the annotation distribution system for the alerting users of the presence 
of annotations. 

25 Figure 7 shows the operation of the system in accordance with the present 

invention, during annotation playback. 

As the most widely used form of hypermedia is the WWW, the invention will be 
illustratively described herein in terms of the WWW, although the concepts and 
general design are fully applicable to any hypermedia system. 
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Figure 1 shows the overall architecture of the system, as implemented for use with 
the WWW. 

A traditional WWW browser has several components, notably a network interface 
which can access documents from remote sites on the network using well-known 

5 protocols such as HTTP, and a document renderer which converts structured 

documents into a form which is understandable by a user. These components are 
widely available for incorporation into extended systems, as for example from the 
FastNet ActiveX component set from NetMasters LLC. To allow the users to 
access the new features of the current invention, several new user-interface objects 

10 are added. Each of these objects is described with the relevant new component. 

Figure 2 shows a layout of a typical instance of this invention, with the new user 
interface objects indicated. 

S3 The invention described in the afore-mentioned U.S. Patent No. 5,838,313 allows 

5=y the user to author dynamic annotations on electronic documents. The present 

jt 15 invention allows the user to author annotations within a program for browsing 

M hypermedia and furthermore exhibits three novel features: the capture and 

[j recording of navigation events, the capability to include multiple documents within 

[ A an annotation, and the capability to insert hyperlinks to existing annotations. 

[JJ On pages 8 and 9 of the afbre-mentioned U.S. Patent No. 5,838,313, a list is 

W 20 provided of "significant events" which are to be recorded in the annotation. 

m Briefly, it is therein stated, in the context of events recorded during recording that 

two types of discrete events are included, the user-activated events and the time- 
sliced events. The user-activated events, of concern herein, have the format 
(event_name, time, parameters). Events were described therein for many typical 
25 user-activated events such as moving a user-interface object on the screen, 

changing the color of some text, erasing a graphic, etc. 

In accordance with the described embodiment of the present invention, the list of 
significant events is extended to include the following navigation events, which are 
common in hypermedia browsers: 

30 - following a hyperlink 





- invoking "forward", "back", "refresh", or "home" browser commands 

- jumping to a document through the use of a history list 

- jumping to a document by invoking a "bookmark" 

- scrolling the document in the viewable area 

5 As with other events described in the afore-mentioned U.S. Patent No. 5,838,313, 

in accordance with the described embodiment of the present invention, each event 
is stored as (event_name, time, parameters). The first four navigation events are all 
stored with the event_name NEW_DOCUMENT, and with a single parameter that 
stores the target document ID. Normally, this ID uniquely identifies the document. 

10 In the case of WWW documents, the Universal Resource Locator (URL) serves as 

the ID. The scrolling events are stored as with the event name SCROLL and have 
one parameter to indicate the scrolling direction (left, right, up or down), and a 
second parameter to indicate the magnitude of the scroll. 

Figure 4 shows the operation of the system in accordance with the present 
15 invention in annotation mode. During the authoring of an annotation, the 

occurrence of a navigation event other than a SCROLL event causes the 
hypermedia browser to load and render a new document in the usual way. The 
recording of the annotation, including the audio recording and the clock, pauses 
while the new document is loading, and resumes when the loading is completed. 
20 This allows multiple documents to be annotated within the same annotation. As 

the clock is paused during loading, times for all events are stored independent of 
the loading time of the various documents accessed. For example, if the user 
causes a MOUSE_MOVE event, the timestamp for this event would be the number 
of milliseconds since the annotation began, minus any time spent downloading. 

25 Another novel feature of the dynamic annotation authoring component is the 

capability to create hyperlinks to existing annotations. The user creates a hyperlink 
by clicking on the target annotation identifier (marked H in Figure 2) in the 
annotation manager box (marked G in Figure 2) both of which are described in the 
next section. The hyperlink is stored as an event with event_name 
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ANNOTATION XlNK, a time, and one parameter giving the identifier of the 
target annotation. 

In order to allow the user to access the features of the dynamic annotation 
authoring system from within the hypermedia browser, two user-interface objects 
5 were added to the traditional browser interface. These are: 

- a browser-mode box with three buttons: browse, annotate and watch (A,B, and 
C in Figure 2) 

- an annotation-mode box with two buttons: scribble and text (D and E in Figure 
2) 

10 A mode box is a collection of options, only one of which can be valid at a time. 

Figure 3 shows the flow of control of the hypermedia browser as determined by the 
two new mode boxes. The browser-mode is normally Browse, in which case the 
annotation-mode box is disabled (shown). In Browse mode, the normal WWW 
operations are in effect. Clicking on the annotate button switches the browser- 

15 mode to Annotate. In this mode (see Figure 4), events (including the navigation 

events listed in the previous section) are captured and stored as in the afore- 
mentioned U.S. Patent No. 5,838,313, depending on the annotation-mode. The 
annotation-mode box is enabled. Clicking on the text button sets the annotation 
mode to Text, and clicking on the scribble button sets the annotation-mode to 

20 Scribble. In both annotation-modes, mouse events are captured. In the Scribble 

mode, MOUSE_MOVE events following a MOUSE_DOWN event, but preceding 
the next MOUSEJJP event cause a line to be drawn. 

In Text mode, no lines are drawn, and MOUSE_DOWN events are used to fix the 
X,Y coordinates for TYPE events. The first TYPE event receives the X,Y 
25 coordinates of the last MOUSE_DOWN event, and successive TYPE events 

receive the same Y coordinate and an incremented X coordinate appropriate for the 
font size. 

As shown in Figure 3, clicking on the browse button returns the browser to Browse 
mode, and ends the annotation. Clicking on the watch button switches the browser 
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to Watch mode in which the current annotation is replayed, as described in section 
"A Means for Playing Dynamic Annotations of Hypermedia." 

A second principal component of the invention is a system for distributing 
annotations. This system includes elements within the hypermedia browser, called 
5 the annotation manager, and elements at a remote server, called the annotation 

server. In general, the system allows the user to store annotations at the annotation 
server. When another user accesses the same hypermedia document, the browser 
checks the server for any annotations on that document. If there are, these 
annotations are made available to the new user for viewing. 

10 In order to allow the user to access the features of the dynamic annotation 

distribution system from within the hypermedia browser two user-interface objects 
were added to the traditional browser interface: 

- a save button (labeled F in Figure 2) 

- an annotation manager box (labeled G in Figure 2), which contains annotation 
1 5 identifiers (labeled H in Figure 2). 

After having authored an annotation and having returned to Browse mode, the user 
has the option of storing it on the annotation server. User interface element F is the 
save button. As shown in Figure 5, pressing the save button invokes a dialog box 
U of the traditional kind, which allows the user to enter a name for the annotation and 

:% s 20 their own name. Once a name is entered, the annotation is written to the local disk 

as an annotation file, as described in the afore-mentioned U.S. Patent No. 
5,838,313. The file is then transferred to the annotation server through the use of 
FTP or a comparable file transfer protocol, along with its name and/or the name of 
the author. An optional feature is that the annotation manager generates a visual 
25 icon representing the annotation, and sends that to the annotation server as well. 

At the annotation server, the annotation file is stored in the annotation store. The 
annotation store can be any kind of traditional database. The annotation file entry 
is indexed by the unique ID of each document included within the annotation. 
These IDs can be extracted directly from the NEW_DOCUMENT events in the 
30 annotation file. 
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As shown in Figure 6, each time a user accesses a document using the hypermedia 
browser, the annotation manager queries the annotation server using the ID of the 
document. If any annotations are stored in the annotation store which have been 
indexed with that ID, the names of these annotations, their authors, and their icons 
if available, are returned to the annotation manager. These names and icons are 
then displayed to the user in the annotation manager box, which is user interface 
element G, each separately, as shown in user interface element H. The user is thus 
made aware of any annotations which may be available for this document. 

There are many protocols for restricting a user's rights to viewing annotations, each 
of which may be useful in different situations. Such protocols and will not be 
particularly described herein, in that the present invention is compatible with a 
number of possible choices of access protocol. 

While the invention disclosed in the afore-mentioned U.S. Patent No. 5,838,313 
allows the user to play dynamic annotations on electronic documents; the present 
invention enables the playing of annotations within a program for browsing 
hypermedia, and extends operation with three new features: the capability to play 
annotations over multiple documents, an enhanced synchronization system to 
handle arbitrary document loading times, and a means for displaying temporal 
hyperlinks to other annotations. 

The user of the system can play previously recorded annotations by clicking on any 
of the annotations listed in the annotation manager box described previously. In 
addition, an annotation author can view the annotation he has just made by 
switching the browser into Watch mode, as shown in Figure 3. The operation of 
the system during playback is shown in Figure 7. 

As described in the afore-mentioned U.S. Patent No. 5,838,313, annotations are 
played by processing each event that has been stored, in turn, while maintaining the 
timing as much as possible. In the case of NEW_DOCUMENT events, the ID of 
the document is passed to the traditional browser modules for loading and 
rendering. Any subsequent events, such as drawing of lines or mouse gestures, are 
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displayed on the new document. In this way, annotations over multiple documents 
can be played. 

A difficulty posed by multiple-document annotation, as opposed to single 
document annotation, is that the time required to load and render a new document 
is not consistent from recording to playback, or even from one playback to another. 
This is particularly true with WWW documents, for which the loading time can 
vary substantially. At playback time, this can cause the recorded audio and events, 
which are on an absolute timeline, to become unsynchronized with the associated 
document. In the present invention, this problem is solved by pausing both the 
recording and playback of annotations during the loading and rendering of new 
documents. Specifically, when a NEW_DOCUMENT event is processed in Watch 
mode, playback is paused until the browser notifies the playback module that 
loading and rendering are complete. Further, the timestamps of all events are 
stored and interpreted relative to the completion of the loading of the most recent 
document. In this way, arbitrary loading times can be tolerated with no loss of 
synchronization. 

As described previously, the present invention allows annotation authors to create 
hyperlinks to existing annotations. These are stored as ANNOTATION_LINK 
events. During playback, ANNOTATION_LINK events are interpreted by causing 
a special "link button" to appear on the browser temporarily. In the described 
exemplary embodiment, the link button appears initially bright red, and then slowly 
fades to white over a period of / seconds. Empirical tests show that five seconds is 
an appropriate value for t. After the link button becomes completely white, it is 
removed. While the link button is visible, the user may click on it. Clicking on the 
link causes the playback of the current annotation to pause, and playing to begin of 
the target annotation, that is, the annotation with the ID stored in the 
ANNOTATION_LENfK event. When the target annotation concludes, the playback 
of the previous annotation resumes at the event where it left off. 

As there may be several annotation hyperlinks followed during a single playback 
session, a stack is required to maintain the state of the playback. Whenever the 
link button is clicked, the current annotation, and the index of the next event in the 



annotation, are placed on the stack. Whenever an annotation concludes, the stack 
is checked to see if there are any outstanding annotations to resume. If the stack is 
empty, the playback is finished; otherwise the annotation on the top of the stack is 
removed, and playback resumes with that annotation. 

While the invention has been described by way of exemplary embodiments, it will 
be understood by one of skill in the art to which it pertains that various changes and 
modifications may be made to the invention without departing from the spirit of the 
invention and that such changes and modifications are intended to be covered by 
the claims following. 
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